Augmented Reality (AR) is a technology that enhances a person's view of the real world with virtual, computer-generated objects such as stationary and/or moving images, text, data, etc., which are projected through a video monitor. Unlike virtual reality (VR), wherein the system places the user in a totally synthetic computer-generated environment, an AR system merges computer-synthesized objects with a user's real world environment, to yield a dynamic environment that is part real and part virtual. Thus, in AR systems, the computer-generated graphics interact with, supplement, and augment the user's interactions and perceptions of a real-world environment.
It is critical to draw a clear distinction between AR environments and VR environments. In a VR environment, the user interacts with (or is “immersed into”) a three-dimensional environment which is entirely computer-generated. Thus, VR systems are described as being virtual, interactive, and immersive. In sophisticated, computationally demanding versions, the VR environment can be a photorealistic, three-dimensional space. But, everything within the VR environment is not real; every object, surface and image is computer generated. Thus, when immersed in a VR environment, the user is incapable of sensing his real world environment.
In contrast, AR systems supplement the real world with virtual, computer-generated objects that appear to co-exist in the same space as the real world. Thus, when immersed in an AR environment, the user is fully aware of his real world environment because the AR-generated objects are layered on top of the user's real world environment. In effect, AR systems link and combine virtual environments with actual environments in a dynamic, interactive, and real time fashion. The AR software generates the virtual objects and their attributes, and juxtaposes images of the virtual objects with images of real objects taken from a real environment. The virtual objects are disposed in registration with specifically defined real world markers.
This last item, registration, is a notable parameter that must be addressed in AR systems, but is wholly absent in VR systems. Because VR systems are entirely computer generated, they are intrinsically self-orienting; the placement of every virtual object in a VR environment can be defined in computer code with respect to another virtual object. In contrast, an AR system must place the virtual objects in proper registration both to other virtual objects generated by the system and to real world objects. Registration of the virtual objects and the real objects in an AR environment is the subject of much on-going research.
A typical augmented reality system includes a display device and a tracking device with associated software housed in a computer. The computer can be of any description—desktop, laptop, hand-held, mobile, or wearable computer. The display device and tracking device may be operationally connected to the computer by any means, such as via a hard-wire connection directly to the computer or to computer network or via wireless connections. The software monitors events detected by the tracking device to determine the present position of the display in the real world and to retrieve virtual objects for use by the user. In order for the display device to present the correct virtual objects, the virtual objects and the real world must be in registration or otherwise synchronized in some fashion. In short, in order for the AR environment to be useful, virtual objects must appear at proper places in the real world so that the user can correctly determine spatial relationships and orientations. For the AR environment to function optimally, registration of the computer-generated graphics must adjust, dynamically and in real time, in response to changes in the user's real world perspective.
Several AR systems are described in the patent literature. A number of these systems specifically address the tracking and registration problem when using mobile AR systems. For example, U.S. Pat. No. 7,274,380, to Naveb et al. (issued Sep. 25, 2007) describes an augmented reality system having a very specific means for determining registration and orientation of the virtual objects with respect to the real objects in the AR environment. Here, the system comprises a video source that resides at a given location and produces an image. At least one encoded marker resides within the image. This encoded marker is formed to store data within a matrix surrounded by a substantially opaque frame. A marker detector is then coupled to the video source. The marker detector is adapted to derive encoded data from the marker that resides within the image. Within the computer is contained a localization processor that is adapted to receive data from the marker detector and to generate data regarding the location and the orientation of the marker. The localization processor then retrieves information from a database that is related to the location and orientation of the marker. The localization processor then making the information retrieved from the database available to a user by presenting the information on the monitor in which the AR environment is presented. The specific attribute of this registration system is that the frame of each marker is formed so as to have an exterior edge string and an interior edge string. The localization processor calculates the relative lengths of the exterior and interior edge strings to establish the presence of a marker within an image. When the marker is detected, its orientation is determined and the appropriate image is retrieved from the database and superimposed on the marker.
A number of other patents and published patent applications address the problem of how to present the virtual objects and the real objects in the proper orientation to each other. See, for example, U.S. Pat. No. 6,765,569, issued Jul. 20, 2004, to Neumann et al. This patent describes an AR system that uses an auto-calibration feature for rendering annotations into images of a real world scene as the camera moves about relative to the scene. U.S. Patent Application Publication No. US 2007/0 038 944, published Feb. 15, 2007, to Carignano et al., describes an augmented reality system having means for gathering image data of a real environment, means for generating virtual image data from the image data, means for identifying a predefined marker object of the real environment based on the image data, and means for superimposing a set of object image data with the virtual image data at a virtual image position corresponding to the predefined marker object. A similar AR system is described in U.S. Patent Application Publication No. US 2007/0 202 472, published Aug. 30, 2007, to Moritz.
See also U.S. Pat. No. 7,204,428, issued Apr. 17, 2007, to Wilson. This object identification system operates using infrared video. In this approach, a coded pattern applied to an object is identified when the object is placed on a display surface of an interactive display. The coded pattern is detected in an image of the display surface produced in response to reflected infrared (IR) light received from the coded pattern by an IR video camera disposed on an opposite side of the display surface from the object. The coded pattern can be either a circular, linear, matrix, variable bit length matrix, multi-level matrix, black/white (binary), or gray scale pattern.
AR software components are commercially available. ARToolworks, Inc., Seattle, Wash., for example, markets a sofware library called “ARToolKit.” The software is made available through a GNU general public license, and through more restrictive licensing terms for commercial applications. ARToolKit is a software library for building AR applications. It is not, itself, an AR application, but rather a tool kit for building AR environments. The ARToolKit software suite addresses one of the key difficulties in developing AR applications: the problem of tracking the user's viewpoint. To know from what viewpoint to draw the virtual imagery, the application needs to know where the user is looking in the real world. The ARToolKit uses physical, distinguishable markers in the real world to calculate the real camera position and orientation relative to physical markers in real time. The software is distributed on-line with the complete source code and allows (in current versions) for single camera position and orientation tracking. The ARToolKit software tracks the virtual objects to the real world markers fast enough to be considered real time.
VR systems for three-dimensional interior design are known and commercially available. See, for example, U.S. Pat. No. 7,277,572, issued Oct. 2, 2007, to MacInnes et al., and the references cited therein. This patent describes a method for generating and rendering a photorealistic, three-dimensional perspective view of a three-dimensional object selectively positioned within a three-dimensional scene.
Commercially available VR software packages for interior design include “Custom Home 3-D Design & Decor” (Sierra, Inc.), “5-in-1 Home Design” (Punch! Software LLC), and “3D Home Interiors” (Broderbund).